Adaptive hybrid activation function for deep neural networks

The adaptive hybrid activation function (AHAF) is proposed that combines the properties of rectifier units and squashing functions. can be used as a drop-in replacement for ReLU, SiL Swish activations deep neural networks evolve to one such functions during training. effectiveness was evaluated on image classification task using Fashion-MNIST CIFAR-10 datasets. evaluation shows with AHAF achieve better accuracy comparing their base implementations use ReLU SiL. A double-stage parameter tuning process training proposed. approach sufficiently simple from implementation standpoint provides high performance network process.